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TITLE: Method for mediating social and behavioral processes in medicine and 
business through an interactive telecommunications guidance system 

Brief Summary Text (27): 

With regard to the prior art, many types of systems have endeavored to provide an 
effective means for providing surveillance over the behavioral modification of a 
patient or client by using a telecommunication link. However, these prior art 
systems have not disclosed an adequate and cost-effective telecommunication network 
that uses a computer in combination with a telephone or other platforms to provide 
positive behavioral based motivational messages and/or questions that are answered 
by a patient or client by means of a dual tone multif requency telephone set or 
other platforms ■ 
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L9: Entry 1 of 1 File: USPT Mar 3, 1998 


DOCUMENT -IDENTIFIER: US 5722418 A 

TITLE: Method for mediating social and behavioral processes in medicine and 
business through an interactive telecommunications guidance system 


Abstract Tent (1) : 

A method for mediating social and behavioral influence processes through an 
interactive telecommunications guidance system for use in medicine and business 
(10) that utilizes an expert (200) such as a physician, counselor, manager, 
supervisor, trainer, or peer in association with a computer (16) that produces and 
sends a series of motivational messages and/or questions to a client, patient or 
employee. (50) for changing or reinforcing a specific behavioral problem and goal 
management. The system (10) consists of a client database (12) and a client program 
(14) that includes for each client unique motivational messages and/or questions 
based on a model such as the trans theoretical model of change comprising the six 
stages of behavioral change (100) and the 14 processes of change (114) , as 
interwining, interacting variables in the modification of health, mental health, 
and work site behaviors of the client or employee (50). The client program (14) in 
association with the expert (200) utilizes the associated 14 processes of change 
(114) to move the client (50) through one of the six stages of behavioral change 
(100) when appropriate by using a plurality of transmission and receiving means. 
The database and program are operated by a computer (16) that at preselected time 
periods sends the messages and/or questions to the client (50) through' use of a 
variety of transmission means and furthermore :j steTleict s a platform of behavioral 
issues that is to be addressed based on a given behavioral stage or goal (100) at a 
given time of day. 


Brief Summary Text (31) : 

The Fuller patent discloses a remote confinement monitoring station and system with 

a central office that provides means for automatic seleat.i.oa of a sfe ^ijLtc^ 

confin,ee. The central office sel ects scheduled or sem-fan3om monitoring calls, to 

..... - , -J : ' 

avoid a^high degree of predictability by the confinee, auto dialing means for 
transmission of a prerecorded or synthesized audio instruction message to the 
confinee, and recording of information received in response to the acts of the 
selected confinee preformed in response to the communicated message. The central 
office has a computer with a telephone line modem, a voice synthesizer, and other 
accessories and displays for automatic recording of data received including a 
visual camera image and breath analyzer results, and can include automatic image 
comparison and violation signal alarming. 


Brief Summary Text (34): 

The Bergeron et al . , patent discloses a method and system for the dispatch of 
resources to remote sites in response to alarm signals. A processor accesses the 
database of, for instance, a field service engineer designated to provide services 
to particular remote sites in response to the alarm signals received from those 
sites. The processor then attempts to establish a telephone connection with the 
field service engineer and provide the engineer with information by means of 
synthesized voice messages. The system may execute remote diagnostic programs and 
determine the results and attempt to communicate with selected resources. The 


http://westbrs:9000/hin/gate.exe?^doc&state=9 lucdn. 16.1 &ESNAME=KWIC&p_Message... 5/23/05 


Record Display Form 


Page 2 of 8 


system has a conventional processor with a database, voice synthesizer, voice 
system and auto dialer. When the system dials and the telephone is answered, the 
system requests an identification code by means of the touchtone buttons before it 
communicates . 

Brief Summary Text (37) : 

The Carroll patent discloses an adaptable electronic monitoring system. The system 
is configured to fit the needs of a particular monitoring or identification 
application by selecting appropriate modules. The system provides for monitoring at 
a central location and communication between the location of the sensed information 
to the processing site by means of a normal telephone communications system. 

Brief Summary Text (39) : 

The D'Alessio patent discloses a method and system for home incarceration using a 
telephone network and voice verification. The system has a control center with a 
process server connected to controllers through a LAN such as an ethernet or wide 
area network. New inmates are added by voice training so that the system can create 
voice templates of fleeted words . A data base of the voice templates and phone 
numbers, work schedules, etc. is created. Calls received are screened by using 
caller ID. Calls to and from the inmate are performed on a predetermined or random 
frequency, the frequency being a function of the patient's behavior. All activities 
are maintained in a log file. 

Detailed Description Text (8) : 

The Dialogic type card or other similar device 16A allows a speech compression 
technique to be used that samples a small difference between speech samples and 
stores a logarithmic function of the difference between the speech samples. This 
technique results in good speech quality at only 3000 bytes per second. The real- 
time clock 16B sets and selects the appropriate time for a particular patient to be 
accessed from the patient^b^'aba'se 12 and the patient program 14. The logic network 
16C provides the logic necessary to determine if a match between the patient or 
employee stored in the database 12 and in the patient or employee program 14 is 
available. The system software 16D provides the algorithms to operate the system 10 
in combination with the logic network 16C. The operating steps of the software 
program are shown in the software flowchart included as FIGS. 2A and 2B. 

Detailed Description Text (10) : 

Both of these signals are applied to the digital signal/ telephone tone signal 
converter 18. The converter 18 includes circuit means to accept and convert the 
digital signals from the computer 16 to telephone tone signals that correspond to 
the patient's or employee's telephone number and a personal identification number. 
The converter 18 can also be designed to include a telephone number redial circuit 
and a random telephone number dialer circuit. The redial circuit allows a patient's 
or employee's busy telephone number to be automatically re-dialed a t „. selectable 
time intervals . The random number dialer allows patient or employee telephone 
numbers stored in the patient or employee database 12 to be randomly selected and 
called. 

Detailed Description Text (33) : 

Another preferred embodiment is the use of a computer-based information metering 
system that uses optical discs 56 as transport and storage media, encrypting to 
protect data and is metered or by other payment means to permit usage by patients 
on a pay-per-view or pay per bit of information basis. The encryption-metering 
device would use digital technology and would be made available through cellular 
phones, wireless cable transmission, modem, interactive television and CD-ROM. 
Information would be distributed in encrypted form to users. After the user browses 
through the menu or index at no charge an ^§g;lects the item needed, the encryption- 
metering device will decrypt the information required, record which data was used, 
by whom and for what issues or subjects, and will permit the user to be billed only 
for the data used. This information would be unreadable or unlistenable until 
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decrypted and users would be charged based on the number of bits of information 
selected . A metering chip or computer board would be used to gauge data use just as 
an electric meter tracks power demand. Information may be retrieved in either full- 
text audio or image form. A decryption program keeps track of how much data is 
decoded and can subtract its costs from a prepaid credit stored on a chip as a form 
of payment. The encryption-metering technology may use a Microsoft Windows. RTM. 
based application or other commercially available software, with familiar graphical 
interfaces and menuing systems to which users are accustomed, and would be 
available on a variety of computer platforms. 

Detailed Description Text (81): 

A novel application of the subject invention is that of extending the prior art of 
physician, manager and counselor behavioral guidance into applying methods utilized 
in the field of education. The subject invention facilitates the application of 
educational principles and research by allowing the daily use of interactive 
quizzes selected by the computer 16 from relevant material to each patient's, 
employee's or client's 50 stage of progress within the aforementioned 
transtheoretical model 100 or other applicable behavioral model. 

Detailed Description Text (101) : 

An additional embodiment is to integrate context-dependent memory into the scheme 
of behavioral reinforcement within the subject invention. Context-dependent memory 
operates on the effect that reinstating the environment in which an event has been 
experienced will bring the memory of that event flooding back. Environmental cues 
have been shown to help to trigger relevant memory trace. In an intervention where 
behavior is reinforced and occurs only within ;a select environmental context, 
replicating the desired environment will triggerT~the desired behavior. 

Detailed Description Text (122) : 

An additional embodiment "is a convertibility feature which permits confi 
subject invention to be in a call-in/call-out mode. The subject inventio 
configured for calling out for behavioral reinforcement and guidance of 
patients or clients 50 in which the client 50 is contacted through the i 
its various embodiments; or, a dial-in system is provided in which the c 
patient contacts the system through various modes which are described in 
embodiments. This is accomplished through the utilization of storage ass 
with the subject invention and a detector which detects when the subject 
is receiving an incoming telephone call. 

Detailed Description Text (124) : 
And yet another embodiment consists of a break-off control or attention circuit 
which may terminate the computer interface aborting to a terminal for direct 
communication with an operator. One appropriate abort capability allows a call to a 
patient or client 50 to go to vocal communication with an operator, nurse, 
physician, expert, or counselor in a live mode in order to establish or reinforce 
rapport, check-up, customer satisfaction, discipline, congratulations, or other 
forms of approval or disapproval all in a personalized manner. Statistical analysis 
and selection can be utilized with respect to patients, employees, or clients 50 
who are provided such "live" communication through the subject invention. 

Detailed Description Text (125) : 

In various operating formats, the patients, employees or clients 50 may be 
variously qualified on the basis of entitlement in order to receive personalized 
intervention. They thus then may be prompted, either through the subject invention 
or through personalized interaction to provide appropriate data. Software 16D can 
be programmed as suggested above, or may be qualified as belonging to an entitled 
set of persons to accommodate specific health, sales, marketing, or management 
objectives. Alternatively, patients or client 50s may be selected based upon 
varying levels of critical need. Where necessary, verification of identity can be 
provided through a personal identification number (PIN) and/or credit card numbers. 
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Detailed Description Text (126) : 

Another distinct operation may involve actuation of a printer 32 to provide 
documents in relation to the operating format, or as for providing award 
certificates or coupons to isolated clients 50 or patients. The subject invention 
in this enhancement thus allows a subset of clients 50 or individuals to be 
isolated with respect to need, infirmity, disease, psychological attribute, or 
change in current condition. Furthermore, the subject invention in this embodiment 
comprises a conventional communication or telephone instrument for voice 
communication means in order to provide vocal operating instructions to an 
individual patient, employee or client 50. Allowing a person to communicate 
directly with said individual. Alternatively, on incoming calls after the patient 
or client 50 has once identified himself with his personal identification number 
(PIN) or credit card, the subject invention includes a cut-through circuit which 
enables the client 50 or patient to directly access a physician, expert 200, nurse 
or counselor for response to questions presented. In the alternative, on incoming 
calls, a member of the aforesaid group could randomly select to interact with 
respondents . 

Detailed Description Text (127) : 

The subject invention thus uniquely allows an individual to verbally communicate 
with patients, employees or clients 50. From the respondent's viewpoint, this makes 
the interaction more interesting in his not knowing when he will hear from a live 
individual for additional guidance. Alternatively, the patient, employee or client 
50 can utilize the subject invention as a "hot line" for emergency or urgently- 
needed reinforcement. In instances where the live expert, manager, 200 or surrogate 
counselor interrupted the incoming call an audible ring or other distinct sound 
could be provided announcing his pending arrival thus allowing physicians, nurses, 
experts and counselors' or their surrogates to selectively interact with patient, 
employee, or client 50 participants in a unique and novel manner which further 
extends the prior art. 

Detailed Description Text (133) : 

Still another preferred embodiment will be the combining of behavioral messages 
which are related in real time and content to each of a plurality of derivative 
secondary interrogatory-containing messages with, preferably, only one of the 
plurality of other tracks comprising a unique selectable responsive message to a 
particular one of the real time related derivative secondary interrogatory messages 
which are chainlinked in content to a particular primary interrogatory message. 
Thus, the stored primary and derivative secondary interrogatory messages and the 
associated responsive messages are spatially related to each other along with 
storage medium tracks in the subject invention in real time and further related in 
content to one another for providing a transitional derivative response to the 
primary interrogatory message dependent on the track selected and the real time of 
selection for providing a conversational real time transitional environment. In 
this manner, the patient, employee or client 50 appears to verbally interact with 
the remotely-located storage medium in the subject invention on an individual 
patient, employee or client 50 basis which may vary from message to message. 

Detailed Description Text (141) : 

An example would be in the field of weight loss. A client 50 could commit to not 
consuming more than 1500 calories a day for five days. In exchange thereof, he 
could select a reinforce r such as going to a motion picture upon the sixth or 
subsequent day which would act as a reward or reinforcer. The announcement and/or 
prerecorded commitment on the patient's or client's part to engage in approved 
behavior during a prescribed interval would be accompanied by a predetermined 
reward which would then be played 'to him upon his achievement of his own or 
negotiated preset goals. By announcing and recording his commitment to his goals, 
the client 50 or patient makes a public semi -commitment to change. The subject 
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invention uniquely allows the client 50 to both choose his reward or reinforcer and 
then to immediately receive confirmation of his achievement of it once he has met 
prescribed goals. This self- reinforcing procedure uniquely extends the prior art 
and allows the client 50 himself to integrate his unique positive behavioral 
reinforcers into his daily lifestyle utilizing computerized telecommunications. 

Detailed Description Text (143) : 

And yet another embodiment would be the application of the behavioral principle of 
stimulus control. In this embodiment the patient, employee or client 50 would 
receive behavioral guidance and reinforcement delivered by the subject invention at 
a fixed time and fixed place on a regular or periodic schedule. A novel 
characteristic among others of the subject invention is that it is flexible and can 
deliver behavioral reinforcement and guidance at any time, 24 hours a day, and to 
any predetermined place which is accessible by remote communications. By selecting 
a predetermined time and place for behavioral reinforcement in conjunction with the 
client 50, the principle of stimulus control can be utilized in conjunction with 
the subject invention in a novel and unique manner. In all prior embodiments, it 
has been assumed that the reinforcement could be applied in either a random or 
fixed time interval. However, in this embodiment both the time and/or the place of 
reinforcement are predetermined and fixed. The client 50 thus associates the place 
of reinforcement with the message of reinforcement within that time interval. In 
other words, if four o'clock and a specific place or room are designated by the 
patient or client 50 and physician or counselor 200, then the daily or periodic 
nature of this prear rang ement acts to amplify the reinforcement of the stimulus of 
the behavioral prompts and cues. 

Detailed Description Text (165) : 

Yet another preferred embodiment would be the provision of a mechanism to provide 
the client 50 or patient with choice as to the mode of presentation of the 
behavioral reinforcement. The exercise of choice is an important cognitive 
principle in guiding human behavior. By providing the option of choice, self- 
control is fostered and self-esteem is enhanced. Inasmuch as the goal of any 
behavioral reinforcement scheme is the fostering of greater positive self-control 
by the patient, employee or client 50, the subject invention uniquely embodies the 
mechanism of choice. Patients, employees, or clients 50 are quizzed or polled as to 
their performance as previously described. In addition they are given choices as to 
the form of the specific intervention. For instance, as part of this embodiment a 
client, employee 50 or patient would be asked whether he preferred to receive 
reinforcement in a visual or auditory mode. By choosing a visual mode he could 
receive text or live or prerecorded video on either a television screen 44, 
computer or telephone screen or video phone 54. Alternatively, the visual text 
might be received on a similar hand-held or portable device 36. If the client 50, 
employee or patient were to choose a portable auditory or visual mode, the subject 
invention would allow him to receive auditory or visual behavioral reinforcement, 
prompts and cues through the use of a wired or wireless telephone or alphanumeric 
beeper device 58 respectively. Upon the client 50 electing to access an auditory or 
visual mode the computer would activate the appropriate storage and/or platform for 
the transmission of behavioral reinforcement accordingly. Alternatively, if the 
client, employee 50 or patient could elect to choose a form of behavioral content 
which would be appropriate for his particular issue. Thus the vital behavioral 
reinforcer of the provision of choice is fostered uniquely by the subject invention 
in a new and novel manner within the patient's, employee's or client's 50 natural 
environment. By providing choice, the client, employee 50 or patient is able to 
select the most personally relevant content and mode, of intervention at the moment 
in order to derive qualitative behavioral reinforcement, explanations, and to 
develop models for making new knowledge fit in a more relevant and meaningful 
structure into what has been previously assimilated. 

Detailed Description Text (179) : 

And yet a further embodiment of the subject invention is the integration of an 
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enhanced graphical interface or hypertext into the text provided by the computer 
16, television screen 44, text phone 36, lap top computer or other similar device. 
Hypertext is an elaborated form of text which is possible through the use of an 
interactive computer interface and may be available on CD-ROMS or other storage or 
related transmission media. Hypertext permits the provision of multiple reinforcers 
in the subject invention. Hypertext is a form of text which allows the patient, 
employee or client 50 to make choices and thus branches accordingly in its 
provision of content. It allows the patient, employee or client 50 to select 
information and proceed according to his own areas of interest and relevance to his 
or her self. The subject invention thus uniquely integrates the behavioral concept 
of choice with a text-driven medium which allows the patient, employee or client 50 
to utilize choice for expanded interest and self-confidence. 

Detailed Description Text (188) : 

By applying human factors and the psychology of perception, the subject invention 
uniquely combines a number of desperate technologies. These technologies for input, 
output, entry, image management and command reduce user disorientation and combine 
to more closely simulate a natural interaction between the physician, counselor, 
administrative or expert 200 and the patient, employee or client 50. Command 
technologies allow the user to tell the subject invention of his choices and 
selections by speaking a command or reply and by directing the system through eye 
movements or gesturing just as pointing devices, like the mouse, have been used in 
the past. Semantic modeling is utilized to enable diagnostic, error recovery, and 
user-cueing procedures. The use of gesture recognition increases the naturalness of 
the interaction between the patient, employee or client 50 and the system and 
further facilitates convenience and ease of use in a novel manner which exceeds the 
prior art. 

Detailed Description Text (208) : 

The specific content of said document would be adapted to each individual patient, 
client or employee 50 based upon his education, gender, age, demographic profile, 
psychological profile and prior response profiles, said educational document and 
text would be further .formulated according to the individual's present behavioral 
stage 100. The patient or employee program 14 would respond to the individual 
response profiles, individual psychological, demographic and other historical data, 
and selectively draw upon generalized educational and motivational data in 
accordance with a behavioral algorithm containing various processes 114 for 
appropriate and timely insights and guidance for each individual all in accordance 
with the predetermined model 100. The patient or employee program 14 would thus 
respond to stored information signals indicative of customized recipient 
information for selecting certain generalized informational and educational prompts 
and cues of said pre-recorded signals indicative of certain segments to be 
retrieved selectively and in a given sequence for compilation purposes all in 
accordance with the predetermined model 100. 

Detailed Description Text (209) : 

All of the aforesaid would be available as text and printed or transmitted by wired 
or wireless telephone, cable, the mail or delivered in person or by video. In 
addition, the aforesaid customized text and graphic communications could be 
accessed by the patient 50 via facsimile (FAX) transmission for behavioral 
guidance. A further variation on this embodiment would be to transmit the text via 
modem and telephone or cable by giving each patient, client or employee 50 a 
telephone number. The system would ask what information he wanted and request his 
fax number along with a menu of additional options. By giving his current FAX 
number through touch tone or voice recognition and selecting menu options by the 
same, the patient 50 would be able to receive a customized hard copy of his 
requested educational and motivational text immediately. Alternatively, the above 
could be sent via modem to the patient's, client's or employee's printer. 

Detailed Description Paragraph Table (2) : 
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TABLE II Correlation Between Behavioral 

Stage and Selected Processes of Change Behavioral Stage Process of Change 

1. Contemplation Intervention Consciousness 

raising Dramatic relief 2. Preparation Self reevaluation Environmental reevaluation 
Helping relationships 3. Action Self liberation Counte rcondi tioning Stimulus 
control Reinforcement management Helping relationships 4. Maintenance 
Countercondi tioning Stimulus control 

Other Reference Publication (2) : 

Nelson, R. Eric and Craighead, W. Edward, " Selective Recall of Positive and 
Negative Feedback, Self-control Behaviors and Depression," Jrnl. Abnormal Psych. 
1977, vol. 86 No. 4, 379-388. 

CLAIMS: 

I. A system for interactive preventative medical guidance and commercial goal 
management comprising: 

a) polling means for creating a database of personalized input data indicative of 
an individual's particular behavioral issue; 

b) evaluation means for determining an individual's temporal behavioral stage from 
said database selected from one of a plurality of behavioral stages; 

(c) mediation means for determining from said evaluation means and said database a 
behavioral routine for changing said selected temporal behavioral stage; 

(d) program means including transmission means for delivering to said individual 
specific content based communication based on said behavioral routine determined by 
said mediation means for changing said selected temporal behavioral stage; and 

(e) feedback means for receiving an individual's response to said content based 
communication wherein said mediation means provides periodic reevaluation of said 
response for determining readjustment of said behavioral routine and said content 
based communication. 

6. The system as specified in claim 5 wherein said behavioral routine for changing 
said selected temporal behavioral stage utilizes guided imagery through prompts and 
ques delivered by said counselor. 

7. The system as specified in claim 5 wherein said behavioral routine for changing 
said selected temporal behavioral stage utilizes systematic des ensi ti zation . 

9. The system as specified in claim 1 wherein said behavioral routine for changing 
said se lected temporal behavioral stage comprises an overlearning technique. 

10. The system as specified in claim 1 wherein said behavioral routine for changing 
said selected temporal behavioral stage comprises application of a generation 
effect . 

II. The system as specified in claim 1 wherein said behavioral routine for changing 
said selected temporal behavioral stage utilizes refresher practice. 

12. The system as specified in claim 1 wherein said behavioral routine for changing 
said selected temporal behavioral stage utilizes contextual variety. 

13. The system as specified in claim 1 wherein said behavioral routine for changing 
said selected temporal behavioral stage utilizes delivery of an increased plurality 
of descriptive examples. 
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14. The system as specified in claim 1 wherein said behavioral routine for changing 
said selected temporal behavioral stage utilizes double-bind quizzes and questions 
to crystallize positive compliance. 

15. The system as specified in claim 1 wherein said behavioral routine for changing 
said selected temporal behavioral stage utilizes an interactive quiz. 

16. The system as specified in claim 1 wherein said behavioral routine for changing 
said selected temporal behavioral stage utilizes time and place shifting in 
delivering said individual specific content based communication. 

17. The system as specified in claim 1 wherein said behavioral routine for changing 
said selected temporal behavioral stage integrates context-dependent memory for 
said individual's particular behavioral issue into said behavioral routine. 

18. The system as specified in claim 1 wherein said behavioral routine for changing 
said selected temporal behavioral stage uses state-dependent memory for said 
individual's particular behavioral issue into said behavioral routine. 

19. The system as specified in claim 1 wherein said behavioral routine for changing 
said selected temporal behavioral stage comprises means of awarding and crediting 
rewards from a predetermined deposit. 
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L8: Entry 1 of 1 File: USPT Jul 21, 1998 


DOCUMENT-IDENTIFIER: US 5782692 A 

TITLE: Time-segmented multimedia game playing and authoring system 
Detailed Description Text (29): 

The multimedia presentation is segmented with respect to specific beginning and 
ending points of segments on the time axis, i.e. there are one or more points of 
time that partition the time axis into time segments. Typically, the multimedia 
presentation can be played either as a whole or in a segment-dependent fashion, for 
instance, when a pause or another media is played in between two time segments, or 
when the user is allowed to select and play an arbitrary time segment. 

Detailed Description Text (32): 

For purposes of this invention, a game, teaching, or instructional tool 
(hereinafter .^^U^-i'm-^ia^^game" ) is a software program that accepts a user's input 
during the time a time-segmented multimedia presentation is played, interprets the 
input as a selection among possible choices, evaluates the choice and provides the 
us^ejr^Mt~h feedback about the -value of the choice s / h e^l^ ^e^le^t ed ■ 

Detailed Description Text (34): 

By linking to the starting. and ending times of each time segment of the 
presentation played, the game layer presents to the user the choices associated 
with each time segment. It then interprets the user's input as a selection of a 
choice, evaluates the selected choice and gives the user a feedback that reflects 
the resulting value. 

Detailed Description Text (36) : 

Referring to FIG. 3b, the explore mode of the multimedia game proceeds in the 
following way. First, the section of the text linked -to the time segment is 
highlighted, and the time segment of the presentation is retrieved and played. If 
there is no user input, the presentation continues playing until its end is 
reached. Then the text is de-highlighted and the presentation proceeds with the 
next time segment. If the user does not interact with the game, all of the time 
segments of the presentation will be played in sequence. The user can, however, 
interrupt the play and choose an arbitrary time segment. This can be realized, for 
instance, by clicking on any desired text section. Then an associated media asset 
(e.g. native speaker's pronunciation of the segment's text) can be played and the 
presentation resumes starting with the selected segment. Other forms of interaction 
with the user are possible, such as playing only one segment at a time or playing 
the time segments in randomly shuffled order. 

Detailed Description Text (37) : 

The control flow diagram in FIG. 3c illustrates the core multimedia game-playing 
algorithm for an example of a game in which a user selects a choice related to the 
text accompanying a multimedia presentation. It depicts the case that time segments 
are played in sequence without interruption. At the beginning of each time segment, 
the linked section of the text track is highlighted, and the choices available to 
the user for this segment are fetched. Then these choices can be presented to the 
user and the time segment of the presentation starts playing. While the time 
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segment of the presentation is played, the multimedia game accepts user input and 
interprets it as a selection of a choice. The game program then fetches the value 
of the selected choice with respect to the current time segment and gives the user 
a sensory feedback of this value. Once the time segment reaches its ending point, 
the text is de-highlighted and the game proceeds with the next time segment. 

Detailed Description Text (39) : 

The user can input in numerous ways, e.g. click with a mouse on a choice within the 
menu, type a key that identifies a choice, select a choice with a joystick or 
pointer, push a software or hardware button while a choice is presented, identify a 
choice using a microphone or any other input device. The input can also be 
continuous, as in the recording of the user's voice. 

Detailed Description Text (41) : 

The evaluation can result in one or more numbers, or an identification of a media 
element (e.g. image or sound), or simply in either a "correct" or an "incorrect" 
feedback, or any combination there of. The value of the user's choice can also 
depend on other factors, such as the history of the user's performance. For 
instance, if several choices are related to the same segment, selecting a choice 
might influence the evaluation of subsequent choices, e.g. the user might be 
required to select the choices in a certain order to receive the maximum score or 
an incorrect choice might decrease the value of the next correct choice. 

Detailed Description Text (65) : 

The SAS is particularly useful for editing a time- segmented presentation to obtain 
a optional multimedia game. As shown in FIG. 12, SAS displays the set of segments 
already specified and ordered according to their start points. It allows the author 
to select one current segment from the sequence of segments. The current segment 
can be then edited. The author can insert a new segment at an arbitrary position in 
the sequence (e.g. before or after the current segment), and delete the current 
segment. 

Detailed Description Text (66) : 

To facilitate editing an individual segment, the system displays the current 
segment's start and end points separately. This way, the author can select which 
one of these "boundary points" s/he wants to edit. The coordinate of the current 
boundary point can be moved with respect to the time axis. Since the end point of a 
segment is closely related to the start point of the next segment, the SAS T user 
interface allows the author to ensure that changes of the boundary point of one 
segment are immediately reflected in the changes of the corresponding boundary 
point of the adjacent segment. The relationship between the two corresponding 
points includes keeping the distance constant (keeping the distance equal to zero 
means that the end of one segment will be always the start point of its neighbor), 
or ensuring that the points remain at least a minimum distance (e.g. to prevent 
overlapping segments) or ensuring that the points maintain at most a maximum 
distance (keeping the points close together can simplify editing of the next 
segment) . 

Detailed Description Text (67) : 

Conventional editing systems allow the author to select and replay a time interval. 
SAS adapts this functionality to segment-based editing allowing the author to 
replay the current segment. However, to specify the boundary point of a lengthy 
segment, it is inconvenient to replay the entire segment. Therefore, SAS allows the 
author to enter a maximum replay time . DELTA. T labeled "delta" on the screen. If 
the start point T.sub.s is being modified, the replay interval is always from 
T.sub.s to (T.sub.s +. DELTA. T) independent of T.sub.s. If the end point T.sub.e is 
being modified, the replay interval is always from (T.sub.e DELTA. T) to T.sub.e 
independent of the value of T.sub.e. The boundary points can be edited using either 
a textual/numeric or graphical interface as described herein. 
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Detailed Description Text ( 68 ) : 

SAS can also be used to simplify the synchronization of the text track with the 
motion video and sound track. It allows the author to enter text into a text field, 
select a portion of the text and assign it to a segment. Similarly, SAS allows the 
author to select a portion of the text track and associate it with choice available 
to the user (or an "event"). 

Detailed Description Text (83) : 

The user can switch between "explore" and "game" mode of interaction as shown in 
FIG. 3a. In the "explore" mode (FIG. 7a), s/he can click within the video window to 
start playing the music video and click again to pause. The next click will resume 
the music video. This way all the segments of the music video can be viewed in 
their natural sequence. Clicking on a phrase in the lyrics field will highlight it, 
say it in native speakers voice and then play the corresponding video segment. The 
user can also click on a segment in the segment bar to play the video starting with 
the selected segment. Both methods of segment selection stop the currently played 
video before continuing. This simple user interface allows the user to replay a 
segment, go to the next or previous segment or choose an arbitrary segment. 

Detailed Description Text (84): 

When the user switches into the "game" mode, the game area is filled with a score 
field a set of tiles that can be selected by the user. The game is a well known 
"f ill-in-the-blanks" game. The tiles are labeled with words extracted from the 
lyrics. In the lyric T s field each of these words has been replaced by a dot 
pattern. The user's task is to fill in as many missing words of the song's lyrics 
as possible. As shown in FIG. 7b, the game program plays the entire sequence of the 
presentation's' time segments and highlights the current segment in the lyrics and 
the segment bar. Clicking on one of the tiles (the only relevant interaction in the 
"game" mode), is evaluated as an attempt to fill the tile's label into the lyrics. 
The user's choices are limited by the time span of the current segment. If s/he 
selects a tile whose label is missing from the current time segment, the score is 
incremented, the label replaces the dot pattern in the lyrics field and the tile is 
taken off the game area. On the other hand, clicking on a tile that does not fit 
into the highlighted segment decreases the score. Removing the tiles reveals the 
singer's picture hidden underneath. 

Detailed Description Text (94): 

As shown in FIG. 10, when a time segment starts, the corresponding dialog 
(typically one or two lines) is displayed on the screen. The user's choices are 
then displayed one at a time. In a "fill-in-the-blanks" game (such as described in 
4.1), the choice can be directly integrated into the current dialog lines. Color 
can be used to distinguish the choice from the original text. According to FIG. 9b, 
each choice within a time segment is displayed for the same length of time (i.e. 
length of the time segment divided by the number of choices) . Another advantage of 
this version is that only one type of input is needed. As only one choice is 
presented at a time, any user input indicates the selection of this choice. User 
input can be provided via designated control button on the game player, or via 
remote buttons or controller unit buttons for interactive TV. This selection is 
then evaluated and its score incremented accordingly. 

Detailed Description Text (109) : 

In the upper left screen area, SAS displays a table that allows the author to 
manage the set of segments. Here, the author can add a segment, delete a segment, 
select a segment, link resources to a segment, and link a text section to a 
segment. The first column of the table contains a field with a bullet marking the 
current segment. The second column holds the segment indices. The third column 
shows the start and end points of each segment. The fourth column contains the 
indices of characters in the text track linked to the segment. The fifth column 
contains the names of additional resources linked to each segment (in our example 
the identification of an associated sound track with the native speaker's 
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pronunciation of the segment's text.) 
Detailed Description Text (121) : 

The third pane of the segment specification area is labeled "adjacent point". It 
specifies how changes of the current boundary point affect the adjacent point. If 
the current boundary point is the start point, the adjacent point is the end point 
of the previous segment. Similarly, the adjacent point of an end point is the start 
point of the next segment. The third pane contains radio buttons "const", "at 
least", and "at most". If none of these buttons is selected, changing the current 
boundary point does not effect the adjacent point. If one of these three buttons is 
highlighted, a field appears to the left of the button. Here the author can specify 
the "delta time span" (in thousandths of seconds), i.e. the distance to be 
maintained in between the current boundary point and the adjacent point. If "const" 
is selected, the delta time span remains always constant, i.e. any change of the 
current boundary point causes the same change in the adjacent point. For instance, 
if delta is zero, then the current boundary point and the adjacent point keep the 
same coordinates during editing. If "at least" is selected, SAS maintains at least 
the delta time span in between the current boundary point and the adjacent point. 
This means that the adjacent point will not be affected if editing increases the 
distance between the points. If editing decreases the distance, SAS may alter the 
adjacent point to maintain the delta time span. For example, this setting can be 
used to ensure that segments do not overlap. Similarly, if "at most" is selected, 
the adjacent point will not be further than delta from the current boundary point. 

Detailed Description Text . (124) : 

The "event table" in the lower left portion of the screen is reserved for the 
module that allows the author to manage the set of choices (or events) . Here, the 
author can add a choice, delete a choice, select the current choice, link value to 
the current choice, link feedback to the current choice, and link selected text to 
the current choice. The table has five columns that show how the choices relate to 
the segments and their intervals in the text track. The left column contains a 
bullet marking the current choice. Each choice must be fully contained within one 
segment; the second and third column identify the section of the text track that is 
linked to the choice (i.e. the indices of the choice's beginning and ending 
characters and the text section itself). The rightmost column contains the 
resources linked to the choice, e.g. the label of a tile to be displayed in the 
game area. 

Detailed Description Text (125) : 

The lower right portion of the screen allows the author to edit the text track. 
Here, he or she can select a section of the text and use the standard text editing 
operations (typing in, cut, copy, and paste) to insert and delete text sections. 
The left field of the text track contains the complete transcript of the text to be 
synchronized. The left field of the text track helps visualize the events by 
replacing the events' text with dot patterns. 

Detailed Description Text (130) : 

The "Author" menu contains commands that edit the segment and event tables (FIG. 
15). The command "Init Segments" clears the entire table (i.e. deletes all 
segments.) The command "New Segment" inserts a new segment either before or after 
the current segment with both boundary points equal to either the start point or 
the end point of the current segment. The command "Delete Segment" deletes the 
current segment. Then, either the following or (if the last segment is deleted) the 
preceding segment becomes the current segment. The command "Update Segment" 
interprets the begin and end of the current selection in the text field as the 
attributes of the current segment and updates these attributes in the segment 
table. 

Detailed Description Text (131) : 

The command "Init Events" in the "Author" menu clears the entire table (i.e. 
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deletes all choices.) The command "Delete Event" deletes the current choice. Then, 
it makes either the following or, if the last choice is deleted, the preceding 
choice the current choice. The command "New Event" inserts a new choice as the last 
choice in the table. The command "Update Event" links the current selection in the 
text track to the current choice. It updates indices of the beginning and ending 
character of the section in the event table and reorders the table if necessary to 
maintain the choices sorted according to the index of the beginning character. The 
command "Associate" provides the author with another way to link a choice with the 
current segment. 

Detailed Description Text (133) : 

The graphical pane of the segment specification area can be also used for editing 
and changing the editing parameters. Dragging the icon of the current boundary 
point changes its coordinate. Since the replay time span must remain constant, this 
operation also changes the position of the "replay limit" icon. Furthermore, 
depending on which button is selected in the "adjacent point" pane, the coordinate 
of the adjacent point might be changed, too. Dragging the "replay limit" icon 
changes the value in the field associated with the highlighted button in the 
"replay" pane. Finally, dragging the icon of the adjacent point changes the "delta" 
time in the "adjacent point" pane. 

Detailed Description Text (137) : 

The Multimedia Game does not need to play the presentation's time segments in their 
natural order. For instance an "Ordering Game" can present the time segments in 
shuffled order and let the user find a predetermined correct sequence. The 
evaluation with the Multimedia Game can be asynchronous, i.e. user's interaction 
with can be recorded and evaluated at later time. This way a teacher can evaluate a 
students' performance and provide motivating feedback. Other tracks' of the 
multimedia presentation can involve other sensory input and output such as smell, 
taste and tactile sensations, as well as activation and analysis of user's neural 
signals. The same extensions of input and output are feasible for the presentation 
and selection of user's choices within the evaluation tool. In addition, 
handwriting recognition can be used for user's input. The presentation and time 
segments can be associated with a variety of other resources. For instance clicking 
on a word in the text track can lead to a dictionary, a thesaurus, an encyclopedia, 
a library catalog or establish a connection with a remote on-line service, initiate 
a search query and display its results. The invention is also independent of the 
presentation of the choices and the evaluation feedback. The choices and the 
feedback can be, for instance integrated into a picture of a board game, such as a 
crossword puzzle or Monopoly. TM. . The extensions of the output described above can 
be used to provide the user with feedback indicating the value of his/her choices. 
The evaluation can involve such feedback as visualization of differences in 
pronunciation patterns and any processes involved in generating speech, such as 
positions of tongue and other parts of human body and air flow patterns or correct 
pronunciation based on the user's voice signature. The difficulty of the tasks can 
be changed by varying the speed of the multimedia presentation in the evaluation 
mode. Alternately tracks can be changed or omitted in evaluation mode, e.g. by 
muting the sound track, hiding the text track or presenting its translation into 
another language. Also the real-time recording of user's interaction with the 
system can be incorporated into either the presentation or the presentation of the 
choices in the evaluation mode. 

CLAIMS: 

1. A method for playing an interactive multimedia game comprising the steps of: 

providing a multimedia presentation comprised of a plurality of recorded media 
tracks with at least a video track that is clocked in accordance with time address 
points of a given time axis; 
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providing a programmed game sequence for play on a player unit, including 
displaying video images of the video track of the multimedia presentation in 
conjunction with receiving user input; 

dividing at least said video track of said multimedia presentation into a plurality 
of presentation segments having respective start and end time address points on the 
given time axis, 

linking each of a plurality of game sequence subunits of said programmed game 
sequence to start and end time address points of respective ones of said video 
track presentation segments; 

displaying game choices stored with each respective one of the game sequence 
subunits in conjunction with displaying respectively linked video track 
presentation segment during a time period delimited by the start and end time 
address points of the respective presentation segment; and 

receiving a user input at any time during the time period of said respective 
presentation segment as a selection of a game choice for the respective game 
sequence subunit linked to said presentation segment, and providing a game sequence 
output determined by said game sequence subunit in accordance with said user input 
selection of said game choice. 

5. A method for authoring an interactive multimedia game for play on a game player 
unit comprising the steps of: 


recording a multimedia presentation comprised of a plurality of recorded media 
tracks with at least a video track that is clocked in accordance with time address 
points of a given time axis; 


providing a programmed game sequence to be operated on the game player unit in 
conjunction with playback of the recorded multimedia presentation, said programmed 
game sequence having user input means for receiving user input entered on the game 
player unit, game sequence display means for displaying game sequence images in 
conjunction with playback of the multimedia presentation, and game sequence output 
means for providing game sequence outputs in response to the user input; 

dividing said video track of the multimedia presentation into a plurality of 
presentation segments having respective start and end time address points with 
respect to the given time axis; 

defining a plurality of game sequence subunits of said programmed game sequence, 
wherein each game sequence subunit is operationally linked to the start and end 
time address points of a respective one of said video track presentation segments, 

defining game choices for each respective one of the game sequence subunits to be 
displayed in conjunction with display of the respectively linked video track 
presentation segment during a time period delimited by the start and end time 
address points for the respective presentation segment; and 

defining game sequence outputs for the respective game sequence subunit in 
accordance with a selection of a game choice made by user input received at any 
time during the time period of the respective multimedia presentation segment, 

whereby the programmed game sequence is authored so that the interactive multimedia 
game can be played on the game player unit by displaying game choices for each of 
the segments of the multimedia presentation and providing game sequence outputs in 
accordance with selections made by user input. 
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